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These tutorials are a simplified 
introduction, and are not sufficient on 
You are responsible for the safety of 
— Roald Amundsen your system. 
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Safety Plan: The Big Picture for Safety po 
= Anti-Patterns for Safety Plans: ery STAND 

e It's just a pile of unrelated documents 3 oo me 

e It doesn't address software integrity "- HAZARDS SAFETY — 

© You don't link to a relevant safety standard / &RISKS GOALS \ 

e It doesn’t link to a security plan | 







MITIGATION & ANALYSIS _ 
= Safety Plan: 


Safety Standard: pick a suitable standard 
Hazards & Risks: hazard log, criticality analysis 
Goals: safety strategy, safety requirements 
Mitigation & Analysis: HAZOP, FMEA, FTA, ETA, reliability, ... 
Safety Case: safety argument 


SAFETY 
CASE 
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Safety Standards 


= Usually “functional safety” (safety functions) 
e IEC 61508 is a generic starting point 
e Many domains have specific standards 


— ISO 26262, EN-50126/8/9, MIL-STD-882, 
IEC 60730, DO-1 78, ... 





m Key elements of a safety standard: 
e Method for determining risk 
— Usually Safety Integrity Level (SIL) 
e SIL determines engineering rigor 
- Analysis techniques 
— Mitigation techniques 
e Life-cycle approach to safety 
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1 [IEC 61508] 
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Safety Goals & Safety Requirements yo 


University 
m Safety Goal: top level definition of “safe” 
e Example: vehicle speed control 
— Hazard: unintended vehicle acceleration 
— Goal: engine power proportional to accel. pedal position 


e Safety strategy: how you plan to achieve goal 
— Example: correct computation AND 
engine shutdown if unintended acceleration 


=m Safety Requirements: 


e Goals at system level; requirements provide supporting detail 


e Supporting requirements generally allocated to subsystems 
— Might include functionality and fail-safe mitigation requirements 
e Examples: 





— Engine torque shall match accelerator position torque curve 
— Pedal/torque mismatch shall result in engine shutdown 
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FMEA: Failure Mode Effects Analysis 


m Idea: Start with component failure; analyze results; identify hazards 
Failure Effects Recommended Action Status 
Done 





Potential Failure Mode 
Use Industrial spec. 


Component 
Resistor R2 Open Triggers Shutdown 
component 
Short Over-current/ Circuit Redesign Open 
potential Fire 
Capacitor C7 Explodes Potential Fire Select different Open 
component 


m Significant limitations for generating hazards 
e “Complex component’ failures are not well behaved 
— Software fails however it wants to fail 
— Integrated circuits are usually highly coupled internally 
e Poor at representing correlated and accumulated faults 
—- E.g., exploding capacitor damaging several nearby components © 2020 Philip Koopman 5 
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HAZard and Operability Analysis (HAZOP) ee 


m Hazard structured brainstorming Guide Word Meaning 
e For each system requirement: NOGRNGr Complete negation of the design 
- Modify with a guide word ee 
— Does the result suggest a hazard? sede Sra ial lad trict 
e Effective starting point, but not ae Senn eae cae 
guaranteed to find all hazards AS WELL AS Qualitative modification/increase 
z Fx ampl as PART OF Qualitative modification/decrease 
: : REVERSE Logical opposite of the design intent 
e When pressure exceeds 6000 psig, relief 2p 
valve shall NOT actuate. VeSEAS Complete substitution 
EARLY Relative to the clock time 
6 ee cit ge to . igen stop 
wath seconds wnen 
: ; LATE Relative to the clock time 
emergency stop is activated. 
BEFORE Relating to order or sequence 


— Alternately: System shall come to a 
complete stop within-5-secends LATE AIS Relating to order or sequence 
when emergency stop is activated. https://goo.gl/KTer9C © 2020 Philip Koopman 6 
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Hazards & Risks ree 


University 





= Hazard: a potential source of injury or damage 
e A potential cause of a mishap or loss event (people, property, financial) 


m= Hazard log 
e Captures hazards for a system 
e HAZOP generates some hazards 
e Others are legacy & experience 


= Risk evaluation 
e Risk = Probability * Consequence 


- Typically determined via a risk table 
e Risk must be reduced to acceptable levels 
— Risk determines required SIL (e.g. “Very High” = SIL 4) 


Consequence 


a eC", 






Se Probability —* 


Very Very Very 
High High High 
Very 
High 






EXAMPLE 


Ve 
RISK Ki 


High 


Very 












RISK 


© 2020 Philip Koopman / 


Carnegie 


Safety Analysis & Mitigation ce 
m Failure Mode Effects Analysis (FMEA) 


e Work forward from fault to mishap 


= Fault Tree Analysis (FTA) 
e Work backward from hazard to causes 
e Strategy: HAZOP identifies fault tree roots 











= Avoid single points of failure 
e If component breaks, is system unsafe? 











e Computational elements fail in worst way lve Lk me Mae | 
COMPONENT FAULTS, 
= Life-critical systems require redundancy SOFTWARE DEFECTS, 
EXCEPTIONS, ETC. 
e Also avoid correlated faults 
e High-SIL software techniques to avoid SW defects Fault Tree 
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= This system is safe because: — on 
structured argument + evidence 
= Incorporates safety plan topics: 
e Methodical identification of hazards 
e Each hazard evaluated for risk 
e Mitigation rigor determined by risk (e.g., SIL) 















Hazards identifi 


SILC 
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softwa 





ae weg ortionment is 
and complete 
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e Analysis rigor determined by risk (e.g., SIL) Sri Somape 


e Safety requirements appropriately cover all hazards 
— Including both accidental faults & malicious faults 

= Example techniques [GSN Standard] 
e Goal Structuring Notation (GSN) http://www.goalstructuringnotation.info/documents/GSN_Standard.pdf 

e Systems-Theoretic Process Analysis (STPA / Leveson) © 2020 Philip Koopman 9 
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Best Practices For Safety Plans as 


= A written Safety Plan including: 
e Hazards + risks 

Safety goals + requirements 

Safety analysis + Mitigation 

e Following a safety standard 

e Resulting ina writtensafety case 

e Independent audit of safety case 












INSTANT DEATH 
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https: //WWW. flickr. Bae su dtoe qucvéteony 1118807 


= Pitfalls: 
e Software safety usually stems from rigorous SIL engineering 
e FMEA can miss correlated & multipoint faults - must use FTA 
e Need to include safety caused by security attacks 
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DANGERS 
INDEXED BY THE NUMBER OF GOOGLE RESULTS FOR 
"DIED IN A ____. ACCIDENT™ 


Nikabe GOOGLE RESULTS 


SKYDIVING 
ELEVATOR, 
SURFING 

SKATEBOARDING | 

CAMPING | 
GARDENING 
ICE SKATING 
KNITTING 
BLOGGING 





https://xked.com/369/ © 2020 Philip Koopman 11 






DETAILS: FiXES AN ISSUE | (THIS UPOATE WILL REQUIRE 
THAT WAS CAUSING RANDOM | RESTARTING YOUR COMPUTER.) | 
LAPTOP ELECTRICAL FIRES 


JRGENT: CRITICAL 


UPDATE AVAILABLE! 





https://xked.com/1328/ 
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